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Ophthalmic statistics note 1 : 
unit of analysis 

SCENARIO 1 

A senior colleague has conducted a study and asks me to analyse 
their data. They give me an excel spreadsheet containing 80 
observations of intraocular pressure (IOP) — 40 of which were 
made after using drug A and 40 of which were made after using 
drug B (see table 1). 

My colleague asks me to find out whether drug A or drug B is 
better in terms of reducing IOP I think back to my medical 
school training and recall something called a t test, which 
I believe may be relevant. Using the internet I discover that there 
is indeed a t test and that the t test can be used to compare the 
means of two groups. I apply the test and feedback the results 
(including a two-tailed p value of 0.0024) to my colleague (see 
online supplementary appendix 1, figure SI for results of ana- 
lysis). The paper is written and submitted. After some time, it is 
returned with several comments made by reviewers. One of these 
questions is whether I have assessed the assumptions made by a t 
test. I must admit to not being entirely clear what this sentence 
means. Using the internet once more, I learn that the p value I 
quoted to my colleague has been calculated using statistical 
theory, but that if this p value is to be regarded as robust or valid, 
my data must adhere to some rules (or assumptions). Assumption 
1 of the t test is that my data follow approximate normality. I 
draw a histogram of my IOP observations and see that it looks 
roughly symmetric, which reassures me somewhat as evidently 
rough normality can be assumed with an approximate symmetric 
histogram (see figure 1). 

Assumption 2 is that the data points are independent. I am 
not familiar with the term independent in this context, but 
using the internet I find that statistically independent means no 
relationship between data points. I have 40 IOP observations 
made on drug A and 40 made on drug B. I email my senior col- 
league to check that the observations are not made on the same 
subjects — for example, the 80 measurements might actually 
relate to measurements on 40 patients each treated with both 
drug A and drug B. If this is the case, then I have measures of 
IOP after treatments A and B on the same patients and it seems 
clear to me that there will be a relationship between IOP mea- 
surements made on the same patient. My colleagues' response 
comes with good and bad news. I am reassured that the subjects 
treated with drug A are not the same as those treated with 
drug B. I am also told, however, that there are actually only 10 
patients on drug A and 10 on drug B. Further discussion reveals 
that my dataset consists of two observations on the right eye 
and two observations on the left eye of each subject (see 
table 2). Suddenly my dataset has revealed a complication that I 
simply hadn't considered. There are multiple observations and 
related observations. Clearly independence isn't adhered to... 
(see online supplementary appendix 1, figure S2 for results of 
analysis of the mean IOP in either the left or the right eye). 

Discussion 

This scenario illustrates what is known as the 'unit of analysis' 
issue. 1 Clinicians treat patients. They may treat the symptoms of 
patients, but the focus is still on the patient. In statistical termin- 
ology, the patient is the sampling unit and should be the unit of 
analysis. Multiple observations may be made on patients, but 
the statistical analysis must not ignore the fact that these obser- 
vations are made on individuals. Failure to do so violates the 
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Figure 1 Histogram of 80 
observations of intraocular pressure 
(overall and by drug group). 




(b) 
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assumption made by the majority of statistical tests that each 
data value is independent. Multiple observations from the same 
patient falsely inflate your sample size, sometimes dramatically 
so, leading to spurious statistical significance. 

Failure to account for unit of analysis issues is common in 
medical research, and certain areas of medicine are more prone 
to such errors than others. 2 For a variety of reasons, including 
stereoscopic vision and greater visual field, mankind has evolved 
with two eyes. While clearly advantageous for the patient, evo- 
lution did not take into account the challenges this might 
present to ophthalmic researchers who unlike their cardiology 
peers routinely face unit of analysis issues. 3 Caution is recom- 
mended when assessing results of any ophthalmic paper that 
simply presents results on eyes. It is important to determine 
how many patients these eyes came from to allow an assessment 



Table 2 80 observations of IOP revealing previously hidden data 
complexities 



of whether or not any patients have contributed fellow eyes so 
presenting unit of analysis issues. 

When designing any ophthalmic research, it is important to 
think carefully about unit of analysis issues at the outset. First, 
consider the question that the research is attempting to answer 
with particular focus on the outcome to be assessed. If the 
outcome is measured at the patient level, for example, a 
quality-of-life measure such as the VF14 or the Sickness Impact 
Profile, there may be a single observation per patient and so no 
unit of analysis issue. 4 5 If the outcome is measured at the 
ocular (eye) level such as logMAR visual acuity or retinal thick- 
ness, then there may be a unit of analysis issue. If the condition 
under consideration is typically unilateral (affecting only one 
eye, perhaps ocular trauma) and the outcome is measured at the 
ocular level, then there will be no unit of analysis issue. Treating 
the patient involves treating only the affected eye and thus mea- 
surements will only be made on that eye. If the condition under 
consideration is frequently bilateral (affecting both eyes, perhaps 
glaucoma) and the outcome is assessed at the ocular (eye) level, 
so that you have two observations per person (one for each eye) 
for some patients, you are likely to encounter unit of analysis 
issues. 







IOP (mm 


IOP (mm 


IOP (mm 


IOP (mm 




Patient 




Hg): right 


Hg): right 


Hg): left 


Hg): left 




ID 


Drug 


eye — 1 


eye — 2 


eye — 1 


eye — 2 


SCENARIO 2 


1 


A 


16 


16 


16 


11 


A senior colleague asks me to analyse data from a clinical trial 


2 


A 


17 


12 


10 


15 


that they have conducted comparing different intraocular lenses 


3 


A 


11 


18 


15 


15 


after cataract surgery. They tell me that the dataset contains data 


4 


A 


17 


13 


19 


19 


on both eyes but that in all cases the patients have received the 


5 


A 


17 


16 


15 


19 


same type of intraocular lens for each eye. They want to know 


6 


A 


12 


14 


16 


13 


whether any change in logMAR visual acuity measured using an 


7 


A 


19 


12 


10 


14 


ETDRS chart at 4 m differs between the two lenses. Table 3 pre- 


8 


A 


14 


15 


16 


15 


sents such data, and it can be seen that there are measures for 


9 


A 


15 


17 


12 


12 


right and left eyes of each patient. These pairs of values are not 


10 


A 


14 


15 


16 


14 


independent of each other because they originate from the same 


11 


B 


16 


16 


14 


15 


patient. 


12 


B 


20 


21 


15 


15 


I decide that to deal with non-independence, I will create a 
summary measure for each patient. 6 I decide in this case to do 


13 


B 


15 


19 


14 


12 


14 


B 


20 


18 


18 


16 


this by selecting one measure per patient. If I select an eye from 


15 
16 


B 


15 


16 


16 


14 


a patient at random, I find there is little evidence of a difference 


B 


19 
16 


19 


19 


15 


between the lenses. However, it is possible to subvert this 


17 


B 


18 


14 


18 


process. Suppose for each patient I choose the eye with the 


18 


B 


16 


15 


12 


19 


greater change if lens A was used, but the eye with the lesser 


19 


B 


18 


16 


14 


15 


change if lens B was used. A t test on the selected eyes now indi- 


20 


B 


16 


17 


18 


18 


cates that there is evidence of an association of greater changes 


in visual acuity with lens A than lens B. However, a t test com- 
paring visual acuity in the fellow eyes shows that lens B leads to 


IOP, intraocular pressure. 
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Table 3 Change in logMAR visual acuity (VA) 



Patient 




Change in logMAR VA: 


Change in logMAR VA: 


ID 


Lens 


right eye 


left eye 


1 


A 


0.08 


0.3 


2 


A 


0.02 


0.2 


3 


A 


0.03 


0.25 


4 


A 


0.17 


0.18 


5 


A 


0.2 


0.37 


6 


A 


0.22 


0.16 


7 


A 


0.04 


0.26 


8 


A 


0 


0.07 


9 


A 


0.12 


0.15 


10 


A 


0.26 


0.24 


11 


B 


0.15 


0.2 


12 


B 


0.02 


0.12 


13 


B 


0.12 


0.04 


14 


B 


0.16 


0 


15 


B 


0.15 


0.1 


16 


B 


0.2 


0.06 


17 


B 


0.12 


0.36 


18 


B 


0.16 


0.2 


19 


B 


0.12 


0.4 


20 


B 


0.12 


0.14 



a greater change in visual acuity than lens A. (See table 4 for a 
summary of the results and online supplementary appendix 1, 
figure S3 for more detailed results of the analysis applying each 
selection criteria). 



Discussion 

Scenario 2 illustrates how single eye studies may be at risk of 
selection bias. Selecting an eye having observed results can 
distort findings. Selecting an eye at random might be appropri- 
ate, but if this option is taken it is wise to repeat the analysis on 
the eye not selected to ensure that the results are consistent and 
that all data were used and not simply wasted. There may be a 
clear rationale for selecting a particular eye — the dominant eye, 
for example, in a study addressing visual acuity or the eye first 
receiving treatment. You may also wish to analyse right and left 
eyes separately to assess whether there is any association 



Table 4 Change in logMAR VA: summary of results 



Selection of eye 



Result of statistical analysis 



1 . The eye with the greater change in 
logMAR VA for each patient with 
lens A 

The eye with the lesser change in 
logMAR VA for each patient with 
lens B 

2. The eye with the lesser change in 
logMAR VA for each patient with 
lens A 

The eye with the greater change in 
logMAR VA for each patient with 
lens B 

3. Select an eye at random for each 
patient 



Lens A is associated with greater 
changes in logMAR VA than lens B 



Lens B is associated with greater 
changes in logMAR VA than lens A 



There is no evidence of difference in 
logMAR VA between patients receiving 
lens A and those receiving lens B 



VA, visual acuity. 



between laterality and the disease of interest. 7-9 An alternative 
to selecting an eye at random might be to create a summary 
measure that is patient dependent. For example, if vision is 
assessed, one might argue that it is the vision in the better eye 
that is of more interest to the study objectives, in which case 
that eye might be selected. Some people advocate calculating a 
summary measure of right and left eyes for each patient — for 
example, the mean IOP being the mean of the right eye pressure 
and the left eye pressure. 3 This might be statistically valid but 
may be clinically unwise. For example, as shown in table 5, 
imagine you have a patient with a right eye IOP of 22 mm Hg 
and a left eye pressure of 32 mm Hg who is treated in a clinical 
trial on ocular hypertension. Suppose there is a reduction of 
10 mm Hg with treatment in each eye. Post-treatment, the right 
eye has a pressure of 12 mm Hg and the left eye has a pressure 
of 22 mm Hg. Taking the mean pressure would give you a post- 
treatment pressure of 17 mm Hg, which might lead you to con- 
clude that you have lowered the pressure sufficiently in both 
eyes. Yet the left eye still has a pressure of 22 mm Hg, indicating 
it may still require treatment to reduce the pressure. If the pres- 
sure in the right eye has reduced by 1 1 mm Hg and in the left 
eye by a pressure of 9 mm Hg, the left eye would now have a 
pressure of 23 mm Hg. Clearly one would expect the reduction 
in fellow eyes to be similar, but marked differences can and are 
observed in trials. 

There are statistical techniques that allow data from both eyes 
to be used and review articles that summarise these. 3 10 11 
These techniques require varying degrees of statistical expertise, 
and a very clear guide on their use has been provided by 
Armstrong. 12 Generalised linear modelling provides a frame- 
work that enables the true structure of the dataset to be 
explored, but this may require more statistical knowledge than 
most practising clinicians wish to acquire. To summarise, studies 
where fellow eyes have been treated similarly require cluster- 
type analytical techniques, while studies where fellow eyes have 
received different treatments require techniques for paired data. 

SCENARIO 3 

You have a glaucoma study where all patients are bilaterally 
affected, patients typically demonstrating symmetric disease. 
Could you and indeed should you now treat one eye with one 
treatment and the other eye with another treatment? 

Discussion 

Some researchers advocate this design arguing that a comparison 
of treatments between fellow eyes makes ideal use of the fact 
that the eyes are similar genetically and environmentally and 
differ only in treatment. By keeping genetic and environmental 
factors constant, the power of a study is increased and fewer 
patients may need to be recruited. Careful thought, however, is 
needed as to whether these arguments are valid for a particular 
scenario. Clearly it is inappropriate to treat fellow eyes with dif- 
ferent treatments if the outcome of the study is assessed at the 
patient level — unless perhaps treatment for each eye is over dif- 
ferent time periods. Even in this case, however, response to the 
first treatment might influence response to treatment in the 
fellow eye. 13 If the outcome is measured at the ocular level, is 
there any possibility that treatment given to one eye might influ- 
ence the outcome in the fellow eye perhaps by a centrally 
mediated response or systemic effects? 14 Patient views should 
also play a key role in choice of study design, anecdotal evi- 
dence indicating strong views for fellow eyes to be treated 
similarly. 
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Table 5 



Baseline IOP 


Baseline IOP 


Mean baseline 


Post-treatment 


Post-treatment IOP 


Mean post-treatment 


right eye (mm Hg) 


left eye (mm Hg) 


IOP (mm Hg) 


IOP right eye (mm Hg) 


left eye (mm Hg) 


IOP (mm Hg) 


22 


32 


27 


12 


22 


17 


22 


32 


27 


11 


23 


17 



IOP, intraocular pressure. 



Figure 2 Ocular unit of analyses 
issues. Note: The scenarios provided 
are fictitious. The principles of 
International Conference on 
Harmonization Good Clinical Practice 
are that each individual involved in 
conducting a trial should be qualified 
by education, training and experience 
to perform his or her respective task. 15 



What is the outcome? 



Is the outcome measured at the patient or 
ocular level? 



Patient level 
(e.g. quality of life, blindness, highest pressure) 



x 



No unit of analysis issue 



Ocular level 
(e.g. intraocular pressure, retinal thickness) 



Potential unit of analysis issue... 

I 



Is the condition bilateral or unilateral? 

i 




Unilateral 



No unit of 
analysis 



Include one eye in the study from the outset 

Note: must present the rationale for the eye selected 



1. Analyse data 
from one eye 

Simplifies 
analysis, but 
prone to 
selection bias 
and wastes 
data 



2. Analyse data from 
both eyes using 
adjusted analyses 

Trickier to analyse, 
but more efficient use 
of all available data 



x 




3. Select one eye to 
analyse at random and 
repeat analysis on 
fellow eye to ensure 
that answers are 
consistent 
Simplifies analysis, uses all 
data, and allows for 
laterality to be explored 



*unless repeat measures on or within an eye 



LESSON LEARNT: UNIT OF ANALYSIS 

Eyes exist in patients, and typically patients have two eyes. This 
presents challenges in the design, analysis and interpretation of 
ophthalmic research. This is an issue that has been highlighted 
previously. It is, however, an issue that still frequently gives rise 
to statistical errors and it is not uncommon for studies that are 
brilliant in terms of methodology and clinical trial design to 
ignore this issue. It is hoped that this paper illustrates the need 
for careful thought about unit of analysis issues whenever 
reviewing, designing or analysing ophthalmic research. As out- 
lined in this paper, figure 2 provides a brief overview of when 
ocular unit of analysis issues may arise and illustrates ways that 
these can be dealt with. The principles of the International 
Conference of Harmonization Good Clinical Practice are that 
each individual involved in conducting a trial should be quali- 
fied by education, training and experience to perform his or her 
respective task. 15 
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